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Abstract 
To answer the call of introducing more Bayesian techniques to organiza- 
tional research (e.g., Kruschke, Aguinis, & Joo, 2012; Zyphur & Oswald, 
2013), we propose a Bayesian approach for meta-analysis with power prior 
in this article. The primary purpose of this method is to allow meta-analytic 
researchers to control the contribution of each individual study to an esti- 
mated overall effect size though power prior. This is due to the consider- 
ation that not all studies included in a meta-analysis should be viewed as 
equally reliable, and that by assigning more weights to reliable studies with 
power prior, researchers may obtain an overall effect size that reflects the 
population effect size more accurately. We use the relationship between 
high-performance work systems and financial performance as an example 
to illustrate how to apply this method to organizational research. We also 
provide free online software that can be used to conduct Bayesian meta- 
analysis proposed in this study. Research implications and future directions 


are discussed. 
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INTRODUCTION 


Meta-analysis is a statistical method of combining findings from multiple studies to get a 
more comprehensive understanding of the population (Hunter & Schmidt, 2004). A simple way 
to combine studies is to calculate the weighted average of correlations between two variables (or 
differences between two treatments) with the sample size being the weight (e.g., Hunter & Schmidt, 
2004). In addition to the averaged effect size, the variation of it can also be investigated to see how 
the effect size changes from one study to another. This method has become increasingly popular 
in management research in recent years. According to a review by Aguinis, Dalton, Bosco, Pierce, 
and Dalton (2011), thousands of meta-analyses have been conducted and published in five major 
management journals from 1982 to 2009. The number of annually published meta-analytically 
derived effect sizes is also expected to keep growing in the future. 

Both fixed-effects and random-effects models have been used in meta-analysis (e.g., Field, 
2001; Hedges & Olkin, 1985; Hedges & Vevea, 1998; Hunter & Schmidt, 2004). Fixed-effects 
models assume the population under study is fixed and homogenous and the finding from each 
study provides an estimate, ideally unbiased or consistent, of the population effect. For example, 
if one wants to study the relationship between job satisfaction and job performance, he or she may 
believe the relationship between the two variables is universal in the population. The differences 
in the reported correlations in identified studies simply result from sampling variation. Random- 
effects models assume the population is variable and heterogeneous and can show different effects 
according to the distinct features that characterize it. For example, for different types of measures, 
research designs, and research samples, the relationship between job satisfaction and job perfor- 
mance can be quite different (Judge, Thoresen, Bono, & Patton, 2001). Therefore, the differences 
in the reported correlations reflect the heterogeneous effect sizes in the population. 

Meta-analysis has been conducted within both the frequentist and Bayesian frameworks al- 
though arguably meta-analysis can naturally be viewed as a Bayesian method in general. The fre- 
quentist methods for meta-analysis can be found in many places such as Hedges and Olkin (1985), 


Hunter and Schmidt (2004), and Rosenthal (1991). Relatively few studies have discussed Bayesian 
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meta-analysis (e.g., Carlin, 1992; Morris, 1992; Smith, Spiegelhalter, & Thomas, 1995; Steele & 
Kammeyer-Mueller, 2008), which has been considered as having several advantages, such as “full 
allowance for all parameter uncertainty in the model, the ability to include other pertinent informa- 
tion that would otherwise be excluded, and the ability to extend the models to accommodate more 


complex, but frequently occurring, scenarios” (Sutton & Abrams, 2001, p. 277). 


Traditional meta-analysis, using either the frequentist or Bayesian approach, typically treats 
each study equivalently. In other words, each study contributes equally to estimated overall ef- 
fect size after considering the weights proportional to sample sizes. However, in many cases, not 
all studies included in a meta-analysis should make equal contribution to the overall effect size; 
treating them equivalently might cause unexpected consequences in meta-analysis. For example, 
strategic management scholars may be interested in the relationships between financial perfor- 
mance and its antecedents, such as human resource management (HRM) practices (Combs, Liu, 
Hall, & Ketchen, 2006) and human capital (Crook, Todd, Combs, Woehr, & Ketchen, 2011). Fi- 
nancial performance can be measured objectively using data from archival data or subjectively 
using survey data. Although both objective and subjective measures are widely adopted in the lit- 
erature, objective information may reflect a firm’s financial status more accurately than subjective 
ratings because the latter involves more cognitively demanding assessments and the informants 
may not always have the best knowledge of the information. Therefore, those using objective mea- 
sures may provide more reliable information of the relationships between financial performance 
and other variables than those based on subjective measures. For another example, due to the dif- 
ficulty of collecting longitudinal data, longitudinal studies often result in a relatively small sample 
size compared with cross-sectional studies obtaining all information from a single source. Even 
though longitudinal designs may help avoid common method bias and reduce inflation of cor- 
relations (Podsakoff, MacKenzie, Lee, & Podsakoff, 2003), their small sample sizes make them 
contribute less to the final result. Instead, the cross-sectional studies with inflated relationships 
may easily dominate the overall effect size because of their large sample sizes. As illustrated in 
the two examples, treating individual studies equivalently may produce potential misleading re- 
sults. However, not much attention has been paid to this issue when estimating overall effect size 
in traditional meta-analysis. To address this research need, this study proposes a Bayesian method 


for meta-analysis that can control the contribution of each individual study through power prior. 
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As we discuss below, this method can allow meta-analysis researchers more flexibility to estimate 
overall effect size by specifying power parameters for individual studies. 

For illustration, the current study focuses on the meta-analysis of sample correlation al- 
though the same method can be applied for other effect size measures. In the following, we first 
demonstrate the use of power prior through a fixed-effects model and then we extend our method 
to random-effect models and meta-regression. Free online software is introduced to carry out 
the Bayesian meta-analysis discussed in this study. The use of Bayesian meta-analysis is further 


demonstrated through a real meta-analysis example. 


BAYESIAN META-ANALYSIS THROUGH POWER PRIOR 


The proposed method is derived based on the Fisher z-transformation of correlation. Suppose 
p is the population correlation of two variables that follow a bivariate normal distribution. For a 
given sample correlation r from a sample of n independent subjects, its Fisher z-transformation, 


denoted by z, is defined as 


% approximately follows a normal distribution with mean 


hes TEP 
ln —— 


227 LG 
and variance ¢ = 3 (Fisher, 1921). 
Meta-analysis of correlation concerns the analysis of correlation between two variables when 
a set of studies regarding the relationship between the two variables are available. Suppose there are 
m studies that report the sample correlation between two variables. Each study reports a sample 
correlation 7; with the corresponding sample size n;. Let z; = 5 In ce denote the Fisher z- 
transformation of 7; and ¢; = 5 In 7 be the Fisher z-transformation of the population correlation. 


Then, ai™ N(G, 9:) with 0; = (n; = ays 


Fixed-effects Models 


We first investigate the situation where the population can be considered as homogeneous 
and, therefore, a fixed-effects model can be used. In this case, the population correlation is 


1+ p 
l—p 
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and z; ~ N(¢, ¢;). 

The use of Bayesian methods requires the specification of priors (e.g., Gelman, Carlin, Stern, 
& Rubin, 2003), which provides a perfect way to conduct meta-analysis. A prior represents infor- 
mation on the population correlation, or its Fisher z-transformation, without any data collection. 
Although a prior is required, it may consist of “no” information through certain types of prior 
such as Jeffreys’ prior (e.g., Gill, 2002; Jeffreys, 1946). Suppose the prior for ¢ follows a normal 
distribution N (Co, Wo) where Go and wo are pre-determined values. For example, ¢ could have a 
prior N(O,1), which means a researcher initially believe the mean value of ¢ is 0, corresponding a 
correlation 0, with variance |. If little to none information is available, the so-called diffuse prior 
can be used by specifying a large variance such as Wp = 10°. 

After collecting data, in the framework of meta-analysis, with the availability of a study, 
one can get a better picture about the population correlation. Bayesian methods provide a way 
to update the information on the population correlation through Bayes’ Theorem. Let z, denote 
the new information on the correlation after Fisher z-transformation and z; ~ N(¢,@). The 


distribution of the population correlation ¢ by combining the prior and the study is 


_ P(O)plald) 
p(¢|21) = p(21) ) 


where p(¢|z1) is called the posterior of ¢ after considering z,. From Appendix A, we can conclude 


that the posterior distribution is also a normal distribution N(¢,, 2,) where 


1 1 
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Therefore, the posterior mean ¢; is the weighted average of prior mean Co and z; where the weights 
are the inverse of variances of prior and data. If the prior is very accurate, e.g., with a small 
variance, the prior mean will exert a big effect on the posterior. For an extreme case, if Wo = 0, the 
posterior mean is Co, which is also the prior mean. On the other hand, if only little prior information 
is available, reflected by large variance of prior, the prior mean has little influence on the posterior. 
For a special case where wo = +00, the posterior mean is z; and therefore, the posterior is fully 


determined by data. 
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The above analysis assumes that z, is fully reliable or the researcher wants to utilize full 
information from z,. However, if, for practical reason, the information in z; is not accurate enough 
(e.g., obtained from a flawed research design), it might distort the posterior. In this situation, 
a researcher might prefer using only partial information from z,;. Using the power prior idea 


developed by Ibriham and Chen (2000), we can get the posterior 


p(¢)[p(2110)]™ 
Pia)” 


p(¢|21, a1) (3) 


where a, is a power parameter. Note that if a; = 0, no information from z, is used while when 
a, = 1, full information of z, is used. Partial information of z; can be utilized by setting a; to 
be a value between 0 and 1. It can be shown (see Appendix B) that the posterior is still a normal 


distribution with N(¢}, W7) where 


I¢,4 4% 
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1 
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Again the posterior mean is a weighted average of the prior mean and z,;. However, note that the 
weight is different from the previous situation because it is related to the power a,. If a, < 1, 
then the weight for z; is smaller than the previous results in Equation|I] This means posterior will 
rely more on the prior. Note that this is equivalent to let z} ~ N(¢, A). The information is passed 
through a normal distribution with the same mean but enlarged variance. 

Suppose without data collection, a researcher’s prior information on ¢ is (0,1). One study 
in the literature reported a correlation 0.5 with the sample size 28 and, therefore, z; = 0.549 with 
variance 0.04. Table|1|shows the posterior mean and variance for ¢ with power a, ranges from 
0 to 1. When a; = 0, the posterior is the same as the prior. When a increases from 0.1 to 1, 
the posterior mean changes towards to z, because more information from z, is included in the 
posterior. Furthermore, the posterior variance is also becoming smaller. In summary, the use of 
power q, influences both the posterior mean and posterior variance and can control the contribution 
of data to the posterior. 

In meta-analysis, data from multiple studies are available. Bayesian methods provide a nat- 
ural way to combine the data together. For example, suppose we have another study with trans- 


formed correlation z2 and its variance 2 as well as the sample size n2. Furthermore, the power a2 
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Table 1 


The influence of the selection of power parameters for a single study 


Data z-transformation Variance 
r, = 0.5 0.549 0.04 
Prior 0 1 
Power Posterior 
Qy Mean Variance 
0 0 1 
0.1 0.392 0.286 
0.2 0.458 0.167 
0.3 0.485 0.118 
0.4 0.499 0.091 
0.5 0.509 0.074 
0.6 0.515 0.063 
0.7 0.520 0.054 
0.8 0.523 0.048 
0.9 0.526 0.043 
1 0.528 0.038 


is used when combining this study. We have already obtained the posterior of ¢ with the first study 
in Equation [3] To get the posterior by combining z2, we can simply view the posterior in Equation 
[3]as a new prior. Then, the posterior of ¢ with both z,; and zg is 


p(¢|z1, a1) [p(z2|¢)]°? 
p(22) , 


p(C|z1, 22,1, a2) = 


From Appendix C, we know posterior distribution is a normal distribution N(¢3, W5) where 
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Clearly, the posterior mean is a weighted average of prior and the two studies. More generally, if 
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we have m studies with z;, n;, and a, the posterior distribution of ¢ is N(¢*,, W*,) with 


a woot Din 6% 
7 es ica 
: 1 
vr = ; 
a ae in 


For illustration, we show the combination of two studies where the first study reported a 
correlation 0.5 with the sample size 28 and the second study reported a correlation 0 with the 
sample size 103. Therefore, z,; = 0.549 with variance 0.04 and z2 = 0 with variance 0.01. A 
diffuse prior (0,100) is used here so that the effect of prior is minimized. Table [2] presents the 
posterior mean and variance for the population correlation with different combinations of power 
for the two studies. First, when no information from the two studies is utilized (a; = a2 = 0), 
the posterior is just the prior. Second, when only the information of Study 1 is fully used (a, = 1, 
Q2 = (0), the posterior mean and variance are essentially the same as the Fisher z-transformation 
and its variance of Study | because of the use of the diffuse prior. Similarly, one can solely use the 
information from Study 2 by setting a; = 0 and a2 = 1. Third, when the information of the two 
studies are used fully (a; = a2 = 1), the posterior mean is about 0.110, the weighted average of 
0.549 and 0 but leaning towards 0 because the second study has a smaller variance. When setting 
Q1 = G2 = 0.5, the posterior mean is still 0.110 but the variance is about 0.016, twice of that when 
Q, = G2 = 1. This is because only partial information is used from the two studies. Similar results 
can be seen from the table when other combination of power is used. In summary, by controlling 


the power parameter, one can control the contribution of each study to meta-analysis. 


Random-effects Models 


When the population is not homogeneous, it is not reasonable to assume that z; has the 
same mean ¢. Therefore, we discuss the random-effects models in the Bayesian framework. A 


random-effects model can be written as a two-level model, 


2=GPG 
G=O+U 


where Var(e;) = @; and Var(v;) = 7. In the model, each z; has its mean ¢; and the grand mean of 


(4) 


¢; is ¢. Based on Fisher z-transformation, z; ~ N(¢;, ¢;). It is often assumed that v; has a normal 
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Table 2 


The influence of the selection of power parameters for combining two studies 


Data z-transformation Variance 
ry = 0.5 0.549 0.04 
ro = 0 0 O01 
Prior 0 100 
Power Posterior 
Q, Qe Mean Variance 
0 O 0 100 
1 0.549 0.040 
0 1 0.000 0.010 
0.1 1 0.013 0.010 
1 0.1 0.392 0.029 
0.5 0.5 0.110 0.016 
O25 - <1, 0.026 0.010 
1 -0,2 0.305 0.022 
0.2 0.8 0.032 0.012 
0.8 0.2 0.275 0.025 
1 1 0.110 0.008 


distribution and, therefore, ¢; ~ N(¢,7). For the random-effects model, we have the fixed-effects 
parameter ¢ and the random-effects parameter 7. The parameter 7 represents the between-study 
variability. The parameter ¢ can be transformed back to correlation that represents the overall 
correlation across all studies. In addition, we can also estimate the random effects ¢;, which can be 
transformed back to correlations for individual studies. 

As for the fixed-effects models, to estimate model parameters for the random-effects models, 
we need to specify priors. In this study, the normal prior N(¢o, Wo) is used for ¢ and the inverse 
gamma prior [G‘(do, Yo) is used for 7 with Co, Wo, do and yo denoting known constants. In practice, 
Co = 0, Wo = 10°, 59 = 107? and yo = 10-3 are often used to reduce the influence of priors. With 


the priors, the conditional posteriors for ¢, 7, and ¢; can be obtained as in Appendix D. Then, the 
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following Gibbs sampling procedure can be used to get a Markov chain for each parameter. 
1. Choose a set initial values for C and rT, e.g., (© = 0 and 7 = 1. 


2. Generate CM 5 = 1,...,m from the normal distribution 


(0) ie 
=O Tee 1 
N : 
1 a? 1 Ls . 
TOTS, FO G, 


3. Generate +“) from the inverse Gamma distribution IG(dy + m/2,7 + pen (og _ 
¢)"]/2). 


4. Generate ¢‘) from the normal distribution 


aa 
Dist ste <o 1 
iN m |, 1 > m 1 


5, Let €© = CM and + = 7 and repeat Steps 2-4 to get (2), r® and ¢) i = 1,...,m. 
The above algorithm can be repeated for R times to get a Markov chain for ¢, 7, and ¢;. It can be 
shown that the Markov chains converge to their marginal distributions after a certain period and 
therefore can be used to infer on the parameters (e.g., Gelman et al, 2003). The period for the 
Markov chains to converge is called the burn-in period. Suppose the burn-in period is k. Then the 
rest of the Markov chain from (k + 1)th iteration to the Rth iteration can be used to get the mean 
and variance of ¢, 7, and ¢;. Because a research is ultimately interested in the correlation, we can 


also get the Markov chain for p = a and for p; = Seance 


To illustrate the influence of power parameters on the random-effects meta-analysis, we con- 
sider a simple example with three studies that report correlations 0.5, 0 and -0.5 with sample sizes 
103, 28 and 103. The Fisher z-transformed data and their variances are given in Table [3} Table [3} 
also reports the estimated overall correlation p and individual correlation p;,2 = 1, 2,3. When the 
power is | for all three studies, the estimated p is approximately 0. Note that the estimated individ- 
ual population correlations for the first and third studies are smaller than the observed ones. This is 
called “shrinkage” or “multilevel averaging” effect of multilevel analysis (e.g., Greenland, 2000). 
The estimated random effects are pulled towards the average effects. If, based on expert opinions 
or other information, we suspect the reported negative correlation could be because of unreliable 
study, we might assign it a different weight. For example, if we give the third study a power 0.1, 


the estimated overall correlation becomes 0.061. Furthermore, if we assign a power 0.01, the over- 
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Table 3 


The influence of the use of power parameters on random-effects meta-analysis‘ 


Data z-transformation Variance 
r, = 0.5 0.549 0.01 
ro = 0 0 0.04 
r3 = —0.5 -0.549 0.01 
Prior 0 100 
Power Posterior mean 
1 2 p Pl p2 P3 
vil 1 -0.002 0.482 -0.001 -0.482 


1 1 O41 0.061 0.476 0.022 -0.305 
1 1 0.01 0.215 0.469 0.099 0.099 


all population becomes 0.215. Therefore, the effect of the observed unreliable negative correlation 


can be controlled through chosen power parameters. 


Selection between fixed-effects and random-effects models. 


The choice of the fixed-effects or random-effects models is often a subjective decision. How- 
ever, we suggest using two methods to assist such a decision. A random-effects model is only 
beneficial when there is significant variation in the population effect sizes. Therefore, for the first 
method, we can test whether the variance of ¢;, 7, is significant. If it is significant, it suggests 
that a random-effects model is preferred. Otherwise, the fixed-effects model, as a special case of 
random-effects models, can be used. Some scholars have argued that the significance test may 
not have enough power to detect variation in population values due to the small number of studies 
(Hunter & Schmidt, 2004). Therefore, we recommend another method to compare the fixed-effects 
model and the random-effects model through the deviance information criterion (DIC, Spiegelhal- 
ter et al., 2002). If the fixed-effects model has the smaller DIC, it is preferred. Otherwise, the 


random-effects model is better used. 
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Meta-regression Models 


When a random-effects model is suggested, it often indicates possible heterogeneity in the 
population. Therefore, predictors or covariates can be identified to explain such a heterogeneity. 
Suppose a set of p covariates are available, denoted by x1, %2,...,%,. Then, a meta-regression 


model can be constructed as below 


BZ=Gte 
Gi = Bir + Pot t+--- + Popitpi + i = B+ V; 
where 3 = (1, Bo,.--, Bp4i)', X& = (1, 21, Lai, ..-, Zpi), and v; ~ N(0,7). If a coefficient 3; is 


(5) 


significant, x, 1s a significant predictor that might be related to the heterogeneity of the population 
correlation. 

To estimate 3 and ¢, we specify the multivariate normal prior for G as N(¢o, Wo) and the 
inverse Gamma prior [G(6o, yo) for 7. Typically, we use the following hyper-parameters for the 
priors: Co = O(p+iyx1, Yo = 10°1 with I denoting a (p + 1) x (p + 1) identity matrix, and 
d09 = yo = 107°. 

With the prior, the conditional posteriors for G, 7, and ¢; can be obtained as shown 
in Appendix E. The conditional posterior distribution of 7 is an inverse Gamma distribution 
T|B,G ~ IG(d9 + m/2, 70 + 724(G — xi3)?/2). The conditional posterior distribution for 


2 is still a multivariate normal distribution 


N (ca + 2X)" (ware + *%8), (wa! + x) 


T vi T 
where 3 is the least square estimate of @ such that 3 = (X’X)~'X’¢ with X = (x1,...,Xm)! as 


the design matrix and ¢ = (¢1, ¢2,.--, Gm)’. The conditional posterior for ¢; is 


AA XB 
oi { By 
di T 
N a | 1 2a 4 ; ' 
o | T b | 


With the set of conditional posteriors, the Gibbs sampling algorithm can be used to generate 


4 


Markov chain for each unknown parameter as for the random-effects meta-analysis. 


SOFTWARE 


To facilitate the use of Bayesian meta-analysis method through power prior, we developed a 


free online program that can be accessed with the URL. http: //webbugs.psychstat.org/ 
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Figure 1. The interface of the online software metacorr 


Bayesian meta-analysis of correlation through power prior 


DATA: Upload or select a file 


Method 1: Upload data (only .txt file allowed) 


Choose File | No file chosen 


Sample size (7): 


Power (a): 
Reliability: 


Covariates (X): 


CONTROL MCMC and OUTPUT 


Number of iterations: 
Burn-in: 


Random effects 


:LIpic 

[Diagnostic plot for all 
:_|Funnel plot 

:_lEmail notification 


‘Run meta-analysis| 


The online program can be used within a typical Web browser. It has 


an interface shown in Figure [1] To use the program, one needs either to upload a new data file or 
select an existing file. Note names of the existing files are shown in the drop down menu. The 
existing file has to be a text file in which the data values are separated by one or more white spaces. 


The first line of the data file will be the variable names, which will be used in the model. 


Next, a user chooses the model to use. For example, the user can choose to use either the 
random-effects model (default option) or the fixed-effects model. Then, information on the model 


can be provided. Both the Correlation and Sample size are required for all analysis, which can 
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be specified using the variable names in the data set. For example, if we use “fi” to represent the 
correlation between financial performance and another variable in the data set, then “fi” should be 
input in the field of Correlation in the interface. Similarly, “n” is used in the Sample size field 
because in the data set, “n” is also the variable name for sample size. In addition, a user can also 
specify the variables for power and covariates used in the model. 

Finally, one can elect to control Markov chain Monte Carlo (MCMC) method and output of 
the meta-analysis. For example, the total number of Monte Carlo iteration and the burn-in period 
can be specified. In the output, one can require the output of the estimates for random effects ¢;, 


DIC, and diagnostic plots for all model parameters including the random effects. If one checks the 


option Email notification, an email will be sent to the user once the analysis is completed. 


AN EXAMPLE 


We use the relationship between high-performance work systems (HPWS) and financial per- 
formance as an example to illustrate the use of Bayesian meta-analysis with power prior. HPWS 
refers to a bundle of human resource management (HRM) practices that are intended to enhance 
employees’ abilities, motivation, and opportunity to make contribution to organizational effective- 
ness, including practices such as selective hiring, extensive training, internal promotion, develop- 
mental performance appraisal, performance-based compensation, flexible job design, and partici- 
pation in decision making (Lepak, Liao, Chung, & Harden, 2006). Strategic HRM scholars have 
devoted considerable effort to studying the influence of HPWS on firm performance in the past 
three decades and consistently found that the use of HPWS is positively related to employee and 
firm performance (Paauwe, Wright, & Guest, 2013). Indeed, recent meta-analyses have demon- 
strated the positive relationships between HPWS and a variety of performance outcomes (Combs 
et al., 2006; Jiang, Lepak, Hu, & Baer, 2012; Subramony, 2009), including employee outcomes 
(e.g., human capital, employee motivation), operational outcomes (e.g., productivity, service qual- 
ity, and innovation), and financial outcomes (e.g., profit, return on assets, and sales growth). The 
purpose of this study is not to compare the results obtained from Bayesian meta-analysis to those of 
previous research. Instead, we just use the research on HPWS as an example and specifically focus 
on the relationship between HPWS and financial performance, which is one of the most important 


considerations of strategic HRM research. 
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We used several search techniques to identify previous empirical studies that examined the 
relationship between HPWS and financial performance. First, we searched for published stud- 
ies in the databases Business Source Premier, Google Scholar, Web of Science, and PsycINFO 
by combining keywords associated with HPWS (e.g., HRM systems, high-performance work sys- 
tems, high-commitment HRM practices, and high-involvement HRM practices) and with keywords 
related to financial performance (e.g., financial performance, profit, return on assets, return on eq- 
uity, Tobin’s Q, and sales growth). Second, we checked the references of previous reviews on 
HPWS (e.g., Combs et al., 2006; Jiang et al., 2012; Subramony, 2009) and added the articles 
that were missed in the database searches. Third, we conducted a manual search of major man- 
agement journals that often publish strategic HRM research, including Academy of Management 
Journal, Journal of Management, Journal of Applied Psychology, Personnel Psychology, Strategic 
Management Journal, Organization Science, Journal of Management Studies, Human Resource 
Management, Human Resource Management Journal, and International Journal of Human Re- 
source Management, to locate studies that were not included in the previous searches. Finally, we 
searched ProQuest Digital Dissertations and conference proceedings for the annual meetings of the 


Academy of Management for unpublished dissertations and conference papers from 2008 to 2013. 


We used four criteria to include studies in the following meta-analysis. First, we only in- 
cluded studies that examined the relationship between HPWS and financial performance at the 
unit-level of analysis (e.g., teams, stores, business units, and firms). Studies conducted at the in- 
dividual level of analysis were excluded from the analysis. Second, we only included studies that 
examined HRM practices as a system. Those examining the relationships between individual HRM 
practices and financial performance were not considered in this research. Third, consistent with 
previous meta-analyses (e.g., Combs et al., 2006; Jiang et al., 2012; Subramony, 2009), we limited 
financial performance to variables indicating financial or accounting outcomes, such as profit, re- 
turn on assets, return on invested capital, return on equity, Tobin’s Q, sales growth, and perceived 
financial performance. Those only reporting the relationships between HPWS and other types of 
outcomes (e.g., employee outcomes and operational outcomes) were not included in the analysis. 
Fourth, the studies need to report at least two pieces of information in order to be included in the 
meta-analysis — correlation coefficient between HPWS and financial performance and sample size. 


This procedure resulted in 56 independent studies that were entered in the following analysis. 
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Before conducting Bayesian meta-analysis, we first corrected the observed correlation from 
each sample for unreliability by following the procedure outlined by Hunter and Schmidt (2004). 
Because HPWS has been considered as a formative construct (Delery, 1998) for which a high 
internal reliability (e.g., Cronbach’s alpha) is not required, we used a reliability of 1 for the measure 
of HPWS. Similarly, we used a reliability of 1 for the objective measures of financial performance 


and used Cronbach’s alpha as the reliability of the subjective measures of financial performance. 


In addition, we consider firm size as a potential moderator of the relationship between HPWS 
and financial performance in order to test the meta-regression model of this study. Firm size is com- 
monly included as a control variable in strategic HRM research, but its moderating effect has rarely 
been explored in either primary studies or a meta-analysis. Two competing hypotheses can be pro- 
posed in terms of its moderating role. On the one hand, some researchers have suggested that 
large organizations are likley to use more sophisticated HRM practices (e.g., HPWS) compared 
with small and mediem enterprises (e.g., Guthrie, 2001; Jackson & Schuler, 1995). As firm size 
increases, firms may also have more advantages such as economy of scale (e.g., Pfeffer & Salan- 
cik, 1978) and thus be more likely to gain benefit from their investment in HRM practices. On 
the other hand, large firms’ financial performance may be more affected by other factors beyond 
human resources (Capon, Farley, & Hoenig, 1990). In this case, the role of HPWS in enhancing 
financial performance may be limited in large firms than in small and medium firms. Taking these 
considerations together, we expect that firm size may moderate the relationship between HPWS 
and financial performance but make no directional prediction of this effect. Firm size is usually 
indicated by the number of employees. Studies with average number of employees greater than 
250 were coded as | (i.e., large firms) and the others were coded as 0 (i.e., small and medium 
firms). 

Table 4] shows the summary statistics of the data used in this example. Among the total of 
56 studies, 46 measured financial performance using the archival data (i.e., objective performance) 
and 10 used subjective measures of financial performance (i.e., subjective performance). In ad- 
dition, 37 studies were coded as large firms and 19 were coded as small and medium firms. The 
observed correlations ranged from 0.01 to 0.52 with sample sizes ranging from 50 to 2136. 

Four power schemes are considered in the meta-analysis. First, every study is given the 


power of 1. In this case, every study contributes to the meta-analysis result fully and equally. This 
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Table 4 


Summary Statistics 


Minmum Mean Median Maximum Standard deviation 


Correlation 0.01 0.22 0.200 0.52 0.13 

Sample size 50 281 191 2136 325 

Reliability 0.74 0.97 1 1 0.07 
Small & Medium: 19 Large: 37 
Objective studies: 46 Subjective studies: 10 


is equivalent to conduct traditional meta-analysis using Bayesian methods. Second, the reliability 
of financial performance of each study is used as power. The reason for this choice is that, if a 
measure is not reliable, only partial information will be used in meta-analysis. Third, two studies 
have sample sizes larger than 1000 (1212 and 2136, respectively). In order to avoid the dominant 
influence of the two studies on the final result, we assign them a power of 0.1 and the rest of studies 
a power of | in meta-analysis. Fourth, arguably a study with a large effect size is more likely to 
be published, which might cause publication bias. Therefore, reducing the influence of the studies 
with large effect sizes might be helpful in reducing publication bias. In this power scheme, we set 
the power at 0.5 for studies with correlations larger than 0.2. For the power schemes 3 and 4, the 


choice of power is rather liberal. A more serious analysis might consider different levels of power. 


Results of Fixed-effects Meta-analysis 


We first apply the fixed-effects meta-analysis model to the example data. Table[5]shows the 
results using the four different power schemes. When every study is assigned the equal power of 1 
(Power 1), the estimated population correlation p is 0.263 (¢ is the Fisher z-transformed estimate). 
If the reliability of financial performance is used as power (Power 2), the estimated correlation is 
about 0.264. However, when the two studies with the largest sample size are assigned a power 
of 0.5 (Power 3), the estimated correlation becomes 0.226. Note the estimated correlation in this 
condition is significantly different from the other two correlation estimates simply based on the 
credible interval estimates. In the observed studies, the correlations for the two study are 0.34 and 


0.45, respectively, both of which are larger than the estimated fixed-effect correlation. When no 
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Table 5 


Results from fixed-effects meta-analysis?” 


Estimate sd CI DIC 

¢ 0.27* 0.008 0.254 0.285 

Power | 184 
p 0.26* 0.007 0.249 0.278 
c 0.27* 0.008 0.255 0.286 

Power 2 175.9 
p 0.26* 0.008 0.249 0.279 
c 0.23* 0.009 0.212 0.247 

Power 3 72.11 
p 0.23* 0.008 0.209 0.242 
¢ 0.22* 0.009 0.205 0.242 

Power 4 77.44 
p 0.22* 0.009 0.202 0.237 


power is used, the two studies pull the estimates close to them because their large sample sizes 
lead to big weights in the estimated correlation. Under the situation where the studies with large 
correlation are assigned a weight 0.5 (Power 4), the estimated correlation is 0.22, which is even 


smaller than the situation of Power 3. This is because the large correlations are downweighted. 


Results of Random-effects Meta-analysis 


Table |6]shows the results from the random-effects meta-analysis. First, the estimated corre- 
lations from the random-effects and fixed-effects methods are quite different (0.23 vs. 0.27) when 
the power is not considered. This is because for the random-effects method, the between-study 
variability is considered. Therefore, extreme studies (e.g., those with unusual large sample sizes) 
are shrunk towards the average. Furthermore, within the random-effects method, there is not much 
difference in the estimated correlation. Second, only for power scheme 4, the estimated correla- 
tion shows notable difference from the rest of the power schemes. The reason is because studies 
with large correlations are downweighted. Third, in all situation, the variance estimate of 7 is 
significant. This indicates there is sufficient variability in the studies to consider a random-effects 


meta-analysis to model the heterogeneity in the population. 
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Table 6 


Results from random-effects meta-analysis”: 


3 


Estimate — sd Cl DIC 
¢ = 0.23* 0.02 0.191 0.269 
Powerl 7 0.016* 0.004 0.01 0.026 -98.45 
p  0.226* 0.019 0.189 0.263 
¢ 6 0,23* 0.02 0.191 0.27 
Power2 7 0.016* 0.004 0.01 0.026 -97.18 
p  0.226* 0.019 0.189 0.263 
¢ 0.228* 0.02 0.19 0.267 
Power3 7 0.016* 0.004 0.009 0.025 -93.99 
p  0.224* 0.019 0.187 0.261 
¢ 0.218% 0.02 0.178 0.259 
Power4 7 0.015* 0.004 0.008 0.024 -85.4 
p  0.214* 0.019 0.177 0.253 


Results of Meta-regression 


19 


From the random-effects meta-analysis, we concluded that the population should be consid- 


ered as heterogeneous.Through meta-regression analysis, we investigate whether the heterogene- 


ity is related to firm size of different studies. Based on the results in Table |7} firm size is not 


significantly related to the individual differences in the population correlation because the slope 


parameter 54 is not significant regardless of the choice of power. Furthermore, the results from the 


first three power schemes are very close. Comparing all four power schemes, power scheme 4 has 


smaller intercept while larger absolute slope. Combined, the results do not suggest the moderating 


effect of firm size on the relationship between HPWS and financial performance. It implies that 


HPWS used in both large firms and small and medium firms are salutary for enhancing financial 


performance. 
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Table 7 


Results from meta-regression?”* 


Estimate — sd Cl DIC 
(intercept) 0.248* 0.034 0.181 0.316 
Power 1 Bo(size) -0.028 0.042 -0.113 0.053 -97.9 
i 0.017* 0.004 0.01 0.026 
G8,Gntercept) 0.249* 0.034 0.181 0.317 
Power 2 Bo(size) -0.029 0.042 -0.113 0.053 -96.63 
T 0.016* 0.004 0.01 0.026 
(2 \Gntercept) 0.245* 0.034 0.179 0.312 
Power 3 Bo(size) -0.027 0.042 -0.111 0.054 -93.5 
T 0.016* 0.004 0.009 0.025 
(G,Gntercept) 0.24* 0.035 0.172 0.31 
Power 4 Bo(size) -0.034 0.043 -0.121 0.048 -84.79 
T 0.015* 0.004 0.008 0.025 


Fixed-effects Meta-analysis, Random-effects Meta-analysis, or Meta-regression? 


Selection among different methods deserves much more investigation. Here, we just illus- 
trate several possibilities using the example above, which certainly have their limitations. In choos- 
ing between fixed-effects and random-effects meta-analysis, one can check whether the variance 
parameter 7 from the random-effects meta-analysis is significant. If it is significant, random-effects 
meta-analysis can be used. Our example showed that random-effects meta-analysis might be pre- 
ferred. For the choice between random-effects meta-analysis and meta-regression, one can focus 
on the significance of the regression coefficients for predictors. If the coefficients are not signifi- 
cant, it might suggest that there is no need to include the proposed predictor in meta-regression. 

We can also directly compare fixed-effects meta-analysis, random-effects meta-analysis, and 
meta-regression using DIC. The model with the smallest DIC indicates it fits the data best. How- 
ever, DIC should only be used to compare models under the same power scheme. The calculation 


of DIC across power schemes would utilize different information and therefore is not valid. For 
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example, under power scheme 1, DICs for the three models are 184, -98.45, and -97.9. This sug- 
gests that the meta-regression fits the current data best. However, there is no given cut-off on when 


a model can be considered as fitting data significantly better. 


DISCUSSION 


The current study presents a Bayesian method for meta-analysis. A unique feature of our 
method is to enable researchers to control the contribution of individual studies included in a meta- 
analysis through power prior. The motivation of this approach comes from the notion that not all 
studies should be treated equivalently when estimating the overall effect size in a meta-analysis. 
By developing an online program and using the example of the relationship between HPWS and 
financial performance, we have shown how to apply this method into management research. In the 
rest of this article, we briefly summarize the example results derived from the method we proposed. 
And then we discuss some implications of this method to meta-analysis in the field of management. 

In the example study, we use four power schemes to assign powers to individual studies 
included in the meta-analysis. As shown in fixed-effects, random-effects, and meta-regression 
models, using the reliability of financial performance as power does not dramatically change the 
results obtained from regular meta-analysis that uses full information provided by each study. This 
is because that only ten studies used subjective measures of financial performance and the use of 
reliability as power would only influence how the ten out of 56 studies contribute to the final results. 
Moreover, the reliabilities for the subjective measures are typically high, so the vast majority of 
the information they provide still contributes to the overall effect size. If one uses another example 
with more subjective measures, the difference in effect size between regular meta-analysis and 
meta-analysis using reliability as power may be more obvious. Either way, our method provides a 
way to evaluate whether reliability influences meta-analysis results. 

When power is used to reduce the influence of two studies with large sample sizes, the 
overall effect size in fixed-effects model becomes significantly different from what is obtained in 
the regular model, and the change is less obvious in random-effects and meta-regression models. 
This is because between-study variability is taken into account in random-effects model, which 
can shrink extreme effect sizes towards the average. However, this does not mean that using power 


to modify the impact of extremely large samples always has a larger impact on fixed-effects model 
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than on random-effects model. It may also depend on the observed correlations of studies with 
large sample sizes. For example, if the correlation of a large sample is similar to the weighted 
average of the rest of the studies, assigning a small power to the large sample may not significantly 


change the overall effect size in either fixed-effects model or random-effects model. 


The influence of power becomes more salient under power scheme 4 where studies with 
correlations larger than 0.2 are assigned a power of 0.5. We argue that this setting can potentially 
be used to deal with publication bias. For example, if we believe the studies with large effect sizes 
are over-sampled, we can assign them power smaller than |. On the other hand, if one believes the 
studies with small effect sizes are under-sampled, power larger than | can also be used. Certainly 


the choice of power needs careful consideration. 


Combined, the example of the relationship between HPWS and financial performance pro- 
vides an initial illustration of our Bayesian approach of meta-analysis. We encourage researchers 
who are interested in this approach to test their data using the developed software and compare the 
results derived from different power schemes. In the following, we shift attention to the implica- 


tions of this method to meta-analysis in management research. 


First of all, we want to make it clear that it is not necessary to apply power prior to all 
meta-analyses. However, if researchers believe certain factors may impact the credibility of re- 
search findings and they can distinguish between more reliable and less reliable studies, we would 
recommend them to estimate the overall effect size with power prior, at least for the purpose of 
comparison. For example, reliability has been commonly used to correct for measurement error 
in meta-analysis (Hunter & Schmidt, 2004). As reliability decreases, the corrected effect size is 
more likely to be enlarged. However, a low reliability often indicates poor quality of a measure, 
which may not accurately reflect the intended construct. Therefore, the study using the measure 
with low reliability may be less likely to represent the true relationship of interest. In addition, 
research design (e.g., cross-sectional vs. longitudinal, single-source data vs. multiple-source data, 
self-report ratings vs. observer ratings, field studies vs. experimental studies) may also affect the 
extent to which a study can provide reliable information of the relationship between two variables. 
Combining study findings without considering the credibility of the information may lead to mis- 
leading results. Although researchers can summarize studies of different characteristics separately 


(e.g., Judge, Colbert, & Ilies, 2004; Oh, Wang, & Mount, 2011; Sin, Nahrgang, & Morgeson, 
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2009), they may still need to combine all studies to yield overall effect sizes that can be used as 
inputs of other analyses, such as meta-analytic structural equation modeling (MASEM; Viswes- 
varan & Ones, 1995). In these cases, researchers may choose to use more information from reliable 
studies through power prior in order to obtain overall effect sizes that can better represent the true 


relationships. 


Bayesian meta-analysis with power prior can also be used to deal with outliers, including 
outliers of observed correlations and outliers of sample sizes. Traditionally, researchers often 
eliminate the most extreme data points to attenuate the influence of outliers on overall effect sizes 
(e.g., Hedges, 1987, Huber, 1980, Tukey, 1960). This is similar to assigning a power of 0 to studies 
considered as outliers and using no information of the eliminated studies in analysis. However, 
rather than deleting the data points completely, researchers can also choose to use only a small part 


of their information by assigning a small non-zero power to those studies. 


One important issue that is out of the discussion of this article is what power value should 
be assigned to each study in meta-analysis with power prior. The method proposed in this study 
cannot determine whether a power prior scheme is realistic or not to reflect the contribution of each 
study to the final results. It is more reasonable for researchers who are familiar with the nature of 
the included studies to make the decisions. The general guideline is to identify the criteria that 
can indicate the credibility of research findings and use it to guide power prior decision in meta- 
analysis. One attempt of this study is to use reliability as a power for studies relying on subjective 
measures, which may reduce the overcorrection for unreliability due to extremely low reliabilities. 
In addition, we recommend that one should always compare the results from the analysis with and 
without power priors to inform the influence of the use of power priors. We encourage more efforts 


to further explore this issue in the future. 


This study can be improved and extended in many ways. First, in both random-effects meta- 
analysis and meta-regression, we assume that the random effects follow a normal distribution. This 
assumption might not be valid when there are extreme values. Further study can incorporate robust 
Bayesian analysis to deal with the problem (e.g., Zhang, Lai, Lu, & Tong, 2013). Second, the 
current study has focused on the development of the method for correlation. However, the method 
can be equally applied to other effect sizes such as mean differences and odds ratios. Third, we 


have discussed the use of DIC for model comparison. The performance of it can be investigate 
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through rigor simulation in the future. 
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With the prior and the information from the first study, the posterior, based on Bayes’ Theo- 
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APPENDIX B 


With the power parameter a, the posterior 
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and ¢} = ¢,/a,. From Appendix A, the posterior is V(B/A, 1/A) where 
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The joint posterior distribution is 
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a= 1 


Now we obtain the conditional posterior distributions. 


First, we get the conditional posterior distribution of 7, which is 


p(TlG;, '& Qi) x 7 50-1—m/2 exp - 270 + _ _ ] 


Therefore, the posterior is inverse Gamma distribution IG(dp + m/2, Yo + [/(G — ©)7]/2). 
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Second, the conditional posterior distribution of ¢ is 


(¢-)? dX(G—-¢)? 


P(CIGis T) & exp 


2Wo QT 
Therefore, the conditional posterior is a normal distribution 
N m | 1 >I m | 1 
tT | wo tT | wo 


Third, the conditional posterior distribution of ¢; is 


eee ea ae 
pave a eae - Soe 7 - 5 | ’ 


which is a normal distribution 


¢ Zi Os 
ae 1 
N 1 | 7 a | | =) 
Tr di wT. di 
APPENDIX E 


The joint posterior distribution for the meta-regression model is 


p(B, T| 26; Ge, 0%) x T) |] patenGls; T) 
T) [Po (24|Ci, bi) P(GIB, 7) 
x |Wol'/ exp [-5(6 — 6) Wo '(B- 60)] 7°! exp - a 
<I 2ng;)“/? exp - s ee (Qnr)~/? exp -> | ! 
The conditional posterior distribution of 7 is 
p(T |B, G;) 0 T~CoF/2)- ex - Jo EG = x; 3) 2) (6) 


Therefore, 7|3,¢; ~ IG(d9 + m/2, 0 + (G: — x;3)?/2). The conditional posterior distribution 
for G is 


(7) 


w(BIr G) «exp | — 5(8 ~ &)'5"(B — Co) ex [EE 


2T 
Let € = (1, ¢2,°°+,Gm)’ be the vector of ¢/s, and B be the least square estimate such that 
@ = (X'X)~-!X’¢ with X = (x;,...,x,,)! as the design matrix. Then the conditional poste- 


rior distribution of 3 is a multivariate normal distribution 


v( (ve = a x 5). (a5! oe ‘ 
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For ¢;, its conditional distribution is 


(2; — Gi)? (G= x6) 
P(GI|B,7) « exp ple [or oe 
a normal distribution 
QZ, XB 1 
N( ~ | 7 7 Oi r) 
ae bo T 
FOOTNOTES 


1. The results are based on a total of 10,000 iterations with the first 4,000 iterations as burn- 
in. 

2. p< 0.05: 

3. Power 1: each study is given a power of 1. Power 2: the reliability of financial perfor- 
mance is used as power. Power 3: the two studies with the largest sample sizes are given a power 


of 0.1. Power 4: studies with correlations larger than 0.2 are given a power of 0.5, otherwise, 1. 


